Batch Normalization Layer is one of the solutions to mitigate this problem that the values are distributed across extremes. As shown in figure 4 above, Batch Normalization Layer performs normalization* on the tensor data obtained by some operations such as multiplication and addition. With this normalization, the gradients of activation function can be well-distributed (without zero-biased gradients) and it will lead to efficient parameter updates.

The sigmoid function is one of the most popular activations functions used for developing deep neural networks. The use of sigmoid function restricted the training of deep neural networks because it caused the vanishing gradient problem. This caused the neural network to learn at a slower pace or in some cases no learning at all. This blog post aims to describe the vanishing gradient problem and explain how use of the sigmoid function resulted in it. Sigmoid functions are used frequently in neural networks to activate neurons.

Training deep neural networks can be a challenging task, especially for very deep models. A major part of this difficulty is due to the instability of the gradients computed via backpropagation. In this post, we will learn how to create a self-normalizing deep feed-forward neural network using Keras. This will solve the gradient instability issue, speeding up training convergence, and improving model performance. Disclaimer: This article is a brief summary with focus on implementation.

The vanishing gradient problem is an issue that sometimes arises when training machine learning algorithms through gradient descent. This most often occurs in neural networks that have several neuronal layers such as in a deep learning system, but also occurs in recurrent neural networks. The key point is that the calculated partial derivatives used to compute the gradient as one goes deeper into the network. Since the gradients control how much the network learns during training, if the gradients are very small or zero, then little to no training can take place, leading to poor predictive performance.

Let me begin this article with a question -- Which of the following sentence makes sense? Its obvious that the second one makes sense as the sequence of the sentence is preserved. So, whenever the sequence is important we use RNN. RNNs in general and LSTMs, in particular, have received the most success when working with sequences of words and paragraphs, generally called natural language processing. Some of the famous technologies using RNN are Google Assistance, Google Translate, Stock Prediction, Image Captioning, and similarly many more.

Vanishing Gradient Problem occurs when we try to train a Neural Network model using Gradient based optimization techniques. Vanishing Gradient Problem was actually a major problem 10 years back to train a Deep neural Network Model due to the long training process and the degraded accuracy of the Model. What happens is that as we keep on adding more and more Hidden layers in The model, the learning speed of the next hidden layers in the model keep on getting faster and faster. Generally, adding more hidden layers tends to make the network able to learn more complex arbitrary functions, and thus do a better job in predicting future outcomes. This is where Deep Learning is making a big difference due to the thousands and millions of hidden layers it has, we can now make sense of highly complicated data such as images, speeches, videos etc and do Speech Recognition and Image Classification, Image Captioning etc. Now when we do Back-propagation i.e moving backward in the Network and calculating gradients of loss(Error) with respect to the weights, the gradients tends to get smaller and smaller as we keep on moving backward in the Network.